You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FlyDSL] M3 index score: NVFP4 decode scorer (score_flydsl_nvfp4) - #7813
Adds the decode block scorer for the MiniMax-M3 lightning indexer over an NVFP4 index cache, next to the existing MXFP4 one (score_flydsl_fp4): same work-map machinery, same score layout, new page format. Exports minimax_m3_index_score_nvfp4_flydsl and minimax_m3_index_score_nvfp4_flydsl_supported. Only gfx950 (MI355X). A follow-up PR adds the prefill scorer and builds on this one.
Why NVFP4 at all, when it is slower than MXFP4. NVFP4 is MiniMax-M3.1's native fp4 KV/index format, and the benchmark rules do not accept a non-native fp4 cache, so the MXFP4 scorer cannot serve the model as it is. The cost is measured in the Performance section below: the NVFP4 arms take 12-15% more time than score_flydsl_fp4 (geomean e4m3/mx 1.125, exact/mx 1.150) and 72 vs 68 B per token.
Page format
One NVFP4 index page is 9216 B: 128 tokens x 64 B of e2m1 codes (two 4-bit codes per byte, low nibble = even element) = 8192 B, followed by 128 tokens x 8 B of e4m3 scales = 1024 B. One scale per 16 elements, no per-tensor scale. Q is e4m3 (bf16 is accepted). The scorer reads the two regions as separate views (k_data [pages, 128, 64] uint8 and k_scale [pages, 128, 8] uint8, each with its own page stride), so a single [pages, 9216] allocation works. Scale bytes must be positive e4m3 and not 0x7F; contents are a caller contract, shapes, dtypes and strides are validated.
Numerics: k_round
k_round="e4m3" (default): K = e4m3_rn_satfinite(e2m1 x scale), the rounding vLLM's NVFP4 indexer applies. With an e4m3 Q it runs on the fp8 MFMA: each CTA builds a table of the 8 e4m3 values of every scale byte in LDS and widens a K dword with v_perm_b32. Scores are bit-identical to vLLM's NVFP4 decode indexer (60 cases, 208,999 live slots, 100% bit-equal on one shared page buffer, live-slot sets identical; table below).
k_round="exact": K = e2m1 x scale, not rounded. The product is exact in f16 and so is an e4m3 Q, so with an e4m3 Q the K dwords are converted straight to f16 pairs and scored on the f16 MFMA; a bf16 Q uses the bf16 MFMA. This is not bit-identical to vLLM. Measured top-k block overlap with vLLM's scores (random-normal K and Q quantized to NVFP4, so a synthetic distribution; rows with at least k live blocks; the e4m3 arm is 1.0 everywhere):
case
scorer arm
k
top-k overlap with vLLM
rows counted
decode S4H1 B2 ctx<=16384
exact
8
0.95312
8
decode S4H1 B2 ctx<=16384
exact
16
0.96094
8
decode S4H1 B2 ctx<=16384
exact
32
0.97656
8
decode S4H1 B2 ctx<=16384
e4m3
8
1.00000
8
decode S4H1 B2 ctx<=16384
e4m3
16
1.00000
8
decode S4H1 B2 ctx<=16384
e4m3
32
1.00000
8
decode S8H4 B4 ctx<=40000
exact
8
0.94727
128
decode S8H4 B4 ctx<=40000
exact
16
0.95850
128
decode S8H4 B4 ctx<=40000
exact
32
0.97217
128
decode S8H4 B4 ctx<=40000
e4m3
8
1.00000
128
decode S8H4 B4 ctx<=40000
e4m3
16
1.00000
128
decode S8H4 B4 ctx<=40000
e4m3
32
1.00000
128
decode S4H4 B2 ctx<=65536
exact
8
0.92969
32
decode S4H4 B2 ctx<=65536
exact
16
0.95898
32
decode S4H4 B2 ctx<=65536
exact
32
0.96191
32
decode S4H4 B2 ctx<=65536
e4m3
8
1.00000
32
decode S4H4 B2 ctx<=65536
e4m3
16
1.00000
32
decode S4H4 B2 ctx<=65536
e4m3
32
1.00000
32
What changes
aiter/ops/flydsl/kernels/minimax_m3_index_score.py (+614/-65): the NVFP4 arms, the e4m3 table and widen helpers (module level, so the prefill scorer reuses them), validation, score_flydsl_nvfp4, index_score_supported_nvfp4. Page id in an SGPR (no waterfall loop in the K loads), causal mask skipped on tiles whose tokens are all visible, f16 scale pair from one u24 multiply, and an NVFP4 depth table for S8H4 batches 3..16 selected with geometry(..., nvfp4=True) / build_work_map(..., nvfp4=True) so MXFP4 work maps do not move. The table gives, on S8H4 (same process, e4m3 arm / exact arm, depth table vs shared map): batch 8 ctx 64K 0.83 / 0.83, batch 16 ctx 32K 0.79 / 0.79, batch 8 ctx 128K 0.95 / 0.92, batch 2..4 unchanged (1.00 +- 0.01).
aiter/ops/flydsl/__init__.py (+10): the two lazy exports.
op_tests/test_flydsl_minimax_m3_index_score.py (+528/-29): see Tests.
No csrc, no tuned CSV: the aiter-config-shape rules do not apply (the diff has no files under aiter/configs).
Performance (decode, vs vLLM)
One MI355X, same process, interleaved, 7 repetitions, two independent runs (r1/r2 columns), 32 shapes: S = query tokens per request, H = index heads, B = batch, ctx = context. mx is aiter's MXFP4 scorer, vllm is the MiniMax-M3 NVFP4 decode indexer of our vLLM tree (models/minimax_m3/amd/ops/nvfp4_index_decode, unmodified). TB/s is effective K-page bandwidth (live pages x 128 x 72 B per token for NVFP4, x 68 B for MXFP4, over the time); K is re-read from the same buffer every iteration, so the small shapes are warm-cache numbers. Ratios are by mean time (below 1 = faster than vLLM). The nvfp4 columns do not include the work-map build, as for MXFP4.
exact/vLLM time ratio: geomean 0.823, worst 0.940, best 0.648, slower than vLLM in 0 of 32 shapes
e4m3/vLLM time ratio: geomean 0.805, worst 0.905, best 0.619, slower than vLLM in 0 of 32 shapes
shapes where exact is slower than vLLM (mean of the two runs):
none
shapes where e4m3 is slower than vLLM (mean of the two runs):
none
Shapes where an NVFP4 arm is slower than vLLM: none in this table. Against aiter's own MXFP4 scorer the NVFP4 arms are slower, as expected for 72 B vs 68 B per token and the extra decode work: geomean e4m3/mx 1.125, exact/mx 1.150 by time.
Raw table: 32 shapes, two runs
S
H
B
ctx
mx us (r1/r2)
exact us (r1/r2)
e4m3 us (r1/r2)
vllm us (r1/r2)
mx TB/s
exact TB/s
e4m3 TB/s
vllm TB/s
exact/vLLM
e4m3/vLLM
4
1
1
4096
3.67/3.63
4.04/4.0
4.07/4.02
4.94/4.93
0.08
0.07
0.07
0.06
0.815
0.820
4
1
1
16384
3.71/3.69
4.1/4.06
4.1/4.08
4.89/4.84
0.30
0.29
0.29
0.24
0.839
0.841
4
1
1
65536
3.77/3.75
4.18/4.14
4.22/4.16
5.09/5.04
1.19
1.13
1.13
0.93
0.821
0.827
4
1
1
131072
4.06/4.03
4.65/4.61
4.7/4.64
5.59/5.51
2.20
2.04
2.02
1.70
0.834
0.841
4
1
2
4096
3.61/3.59
4.0/3.94
4.02/3.98
4.96/4.94
0.15
0.15
0.15
0.12
0.802
0.808
4
1
2
16384
3.74/3.71
4.12/4.09
4.15/4.11
5.03/4.98
0.60
0.57
0.57
0.47
0.820
0.825
4
1
2
65536
4.04/3.97
4.66/4.61
4.71/4.66
5.59/5.58
2.22
2.04
2.01
1.69
0.830
0.839
4
1
2
131072
4.66/4.64
5.76/5.74
5.55/5.51
7.31/7.36
3.83
3.28
3.41
2.57
0.784
0.754
4
1
8
4096
3.76/3.74
4.16/4.11
4.16/4.14
4.9/4.86
0.59
0.57
0.57
0.48
0.847
0.850
4
1
8
16384
4.04/3.98
4.66/4.59
4.68/4.62
5.42/5.43
2.22
2.04
2.03
1.74
0.853
0.857
4
1
8
65536
7.06/7.05
8.03/7.95
7.89/7.84
10.87/10.84
5.05
4.73
4.80
3.48
0.736
0.725
4
1
8
131072
12.02/11.93
13.01/12.96
12.49/12.45
16.19/16.21
5.95
5.82
6.05
4.66
0.802
0.770
4
1
32
4096
3.94/3.86
4.65/4.58
4.68/4.62
5.5/5.46
2.28
2.04
2.03
1.72
0.842
0.849
4
1
32
16384
6.95/6.95
8.25/8.21
8.07/8.0
11.15/11.18
5.13
4.59
4.70
3.38
0.737
0.720
4
1
32
65536
21.63/21.75
23.35/23.15
23.17/23.53
27.9/27.76
6.57
6.50
6.47
5.43
0.835
0.839
4
1
32
131072
42.51/42.1
45.95/46.38
46.44/46.7
51.44/51.52
6.74
6.54
6.48
5.87
0.897
0.905
8
4
1
4096
3.74/3.71
4.42/4.4
4.21/4.17
5.14/5.1
0.07
0.07
0.07
0.06
0.861
0.818
8
4
1
16384
3.89/3.85
4.58/4.56
4.35/4.32
5.14/5.12
0.29
0.26
0.27
0.23
0.891
0.845
8
4
1
65536
3.97/3.98
4.74/4.73
4.49/4.48
5.3/5.3
1.12
1.00
1.05
0.89
0.893
0.846
8
4
1
131072
4.22/4.28
5.03/4.98
5.02/4.99
5.82/5.79
2.10
1.89
1.89
1.63
0.862
0.862
8
4
2
4096
3.8/3.78
4.5/4.47
4.27/4.24
5.13/5.09
0.15
0.13
0.14
0.12
0.878
0.833
8
4
2
16384
3.91/3.89
4.62/4.57
4.39/4.35
5.21/5.19
0.57
0.51
0.54
0.45
0.884
0.840
8
4
2
65536
4.27/4.29
4.97/4.94
4.99/4.97
5.83/5.79
2.08
1.90
1.89
1.62
0.853
0.857
8
4
2
131072
5.54/5.52
6.72/6.66
6.13/6.09
7.66/7.64
3.22
2.82
3.09
2.47
0.875
0.799
8
4
8
4096
3.94/3.93
4.65/4.62
4.43/4.4
5.12/5.1
0.57
0.51
0.53
0.46
0.907
0.864
8
4
8
16384
4.27/4.36
4.98/4.92
5.0/4.89
5.74/5.69
2.06
1.91
1.91
1.65
0.866
0.865
8
4
8
65536
9.36/9.22
11.61/11.62
10.75/10.73
12.37/12.35
3.84
3.25
3.52
3.05
0.940
0.869
8
4
8
131072
13.29/13.35
16.25/16.19
14.52/14.51
23.47/23.43
5.35
4.66
5.20
3.22
0.692
0.619
8
4
32
4096
4.37/4.31
5.0/5.01
5.06/5.03
5.7/5.69
2.05
1.89
1.87
1.66
0.879
0.886
8
4
32
16384
8.13/8.11
9.3/9.31
8.69/8.67
12.32/12.14
4.39
4.06
4.35
3.09
0.761
0.710
8
4
32
65536
23.19/23.24
26.39/25.93
25.93/25.9
40.33/40.41
6.14
5.77
5.83
3.74
0.648
0.642
8
4
32
131072
48.06/48.13
53.08/52.74
52.28/52.28
81.64/81.74
5.93
5.71
5.78
3.70
0.648
0.640
Tests
python3 op_tests/test_flydsl_minimax_m3_index_score.py (about 20 to 40 s on one MI355X; exits non-zero on any failed check): the existing MXFP4 checks, then the NVFP4 ones, then the sweep tables. The NVFP4 part follows the aiter op-test shape: no pytest in the added code (check_nvfp4_* functions run by the file's own runner, as for MXFP4), @benchmark with run_perftest, checkAllclose, TFLOPS and TB/s per candidate, a markdown table, and get_gfx_runtime() gating in main(). The torch oracle dequantizes the cache itself and a torch writer produces the pages, because this change has no NVFP4 write kernel. Matrix: oracle match on page and tile boundaries, ragged and empty requests; bit-exact on exact-sum data; tail pages; scale range and saturation; the depth table; e4m3 Q equals bf16 Q on the exact arm; split vs single-region views; repeat launches bitwise identical; the default k_round is "e4m3"; compiled kernels are not shared between modes; rejections; both k_round values and both Q dtypes; and a sweep through every non-CP launch geometry the selector can reach (10 geometry points x bf16/e4m3 Q x e4m3/exact K, 40 checks, about 21 s on its own, run by check_fp4_every_geometry_matches_oracle, the same function the MXFP4 builder calls). A wider regression matrix (183 NVFP4 decode checks, hundreds of shapes, bit comparison against vLLM) was run outside the PR on this tree; it is not part of the diff.
Checked on this tree with flydsl 0.5.0 (the version requirements.txt pins), gfx950:
183 NVFP4 decode checks of the wide matrix pass; the 100 existing MXFP4 checks pass; the 76 sha256 hashes of MXFP4 decode and prefill outputs are identical to origin/main.
NVFP4 decode outputs (both arms, both Q dtypes, 128 cases) are byte-identical to the development branches this PR was cut from.
e4m3 arm vs vLLM: 60 cases, 208,999 live slots, 100% bit-equal.
Dependencies and scope
flydsl==0.5.0 as pinned, gfx950 only. Not in this PR: the NVFP4 index write kernel (the tests write pages with torch), the prefill scorer (next PR), ATOM integration.
PR Self-Review
Reviewed revision
499b0e0 (head after the second 11-15 review round; the full sha is the PR head)
Behavior and intent
Adds score_flydsl_nvfp4 and index_score_supported_nvfp4: decode block scoring over an NVFP4 index cache (9216 B pages: e2m1 codes plus one e4m3 scale per 16 elements). Before this PR aiter could only score the MXFP4 cache, so the MiniMax-M3 NVFP4 index cache had no aiter scorer. The default k_round="e4m3" arm is bit-identical to vLLM's NVFP4 decode indexer; k_round="exact" keeps K unrounded and is not bit-identical. The predicate returns False for any k_round the scorer rejects, including None.
Validation and unverified scope
Run: python3 op_tests/test_flydsl_minimax_m3_index_score.py on one MI355X (gfx950, flydsl 0.5.0): exit 0 on the head above. Re-checked after the review fixes: 76 sha256 hashes of MXFP4 decode and prefill outputs identical to origin/main, and the NVFP4 decode output hashes (both arms, both Q dtypes) identical to the pre-fix tree. Run before the review fixes, on the same kernel code: 183 NVFP4 decode checks and 60 cases / 208,999 live slots 100% bit-equal to vLLM for the e4m3 arm (not re-run, the fixes touch no kernel code). Unverified: gfx942 and other flydsl versions (gfx950 only by design), real selector/model data (synthetic random-normal K and Q), the "exact" arm's agreement with vLLM beyond the top-k overlap table. The hgemm CI failure (test_flydsl_hgemm.py) is outside this diff.
Re-run at the revision above after the review fixes (3 commits on 80ff1e7: docstrings and comments, test comments, and one refactor folding fp8_dwords_to_f16 into fp8_dwords_widen(raw, lo, dwords, dtype)):
python3 op_tests/test_flydsl_minimax_m3_index_score.py --ctx-len 8192 on one MI355X (gfx950, flydsl 0.5.0): exit 0, 153 checks PASS, 0 FAIL, before and after the refactor (at 6e6f677).
After the 11-15 review round (6a9e5a8: comment and docstring fixes, f16_arm named flag, docs of nvfp4=True on the map builders; 9a5d21b: the NVFP4 every-geometry sweep, expect_value_error on the int8 and wide-stride scale cases, single_region_views shared): the default run on one MI355X passes 193 checks, 0 FAIL, including the new sweep of 40 checks (10 non-CP geometry points x bf16/e4m3 Q x e4m3/exact K); the sweep alone takes about 21 s. The kernel change in that round is only the f16_arm flag (equal to the expression it replaces at that branch) and comments; the ISA and the performance tables were not re-measured after it.
Second 11-15 review round (c4c66f3: the dead bf16 branches of the f16 arm replaced by fx.Float16, the LDS/barrier comments, score_flydsl's public docstring; 499b0e0: one format-neutral check_fp4_every_geometry_matches_oracle for both builders instead of an NVFP4 wrapper): the final ISA of 56 NVFP4 e4m3-Q kernels (both k_round, 28 of them the f16 arm), compiled without a GPU before and after, is byte-identical by sha256; the default run of the op test on one MI355X passes 193 checks, 0 FAIL, before and after (100 MXFP4 and 93 NVFP4 checks, same names). The performance tables were not re-measured.
The refactor does not change code generation: the final ISA (FLYDSL_DEBUG_DUMP_ASM) of all 65 kernels the test builds, including the NVFP4 e4m3-Q arms for both k_round values, is byte-identical before and after (diff -rq).
black and ruff clean on both changed files (ruff 0.17.3 locally; CI pins 0.16.0, which I did not run).
Not re-run: the 183-check wide matrix and the vLLM bit comparison; the fixes touch no kernel numerics (ISA identical), but those runs were not repeated.
PR-specific question
Before this PR every NVFP4 check in the changed test fails at the first call (score_flydsl_nvfp4 does not exist). Beyond that, the checks pin things a plain oracle match would not: bit-exact scores on exact-sum data, tail pages, scale range and saturation, empty and ragged requests, e4m3 Q equals bf16 Q on the exact arm, repeat launches bitwise identical, the k_round default and the predicate for k_round=None, and rejection of bad shapes/strides/dtypes. The tuned work-map check fails if the MXFP4 exact-grid check in _validate_map is removed. No canonical test covers the NVFP4 page format; the existing MXFP4 test only rejects such a cache.
Changed op tests
op_tests/test_flydsl_minimax_m3_index_score.py (+528/-29): the canonical index-score test, extended in place (no new file) with the NVFP4 oracle, correctness matrix and benchmark; the MXFP4 checks check what they checked before, apart from the arch gate (now get_gfx_runtime()); single_region_views and the every-geometry check (check_fp4_every_geometry_matches_oracle, called by both the MXFP4 and the NVFP4 builder) are now shared with the NVFP4 side. It is not redundant with any other test: no other test builds NVFP4 pages or calls the NVFP4 scorer.
Decode block scorer over an NVFP4 index cache: the same e2m1 data view as the
MXFP4 path, but one e4m3 scale byte per 16 elements (8 per token, 9216 B per
128-token page) and no per-tensor scale; Q is e4m3 or bf16.
Two numerics, selected by k_round:
- "e4m3" (default): K = e4m3_rn_satfinite(e2m1 * scale) through a per-CTA LDS
table of the 8 e4m3 values of every scale byte, scored on the fp8 MFMA with
an e4m3 Q; bit-identical to vLLM's NVFP4 indexer.
- "exact": K = e2m1 * e4m3 scale, unrounded. The product is exact in f16 and
so is an e4m3 Q, so with an e4m3 Q the K dwords are converted straight to f16
pairs and scored on the f16 MFMA; a bf16 Q uses the bf16 MFMA. Not
bit-identical to vLLM, which rounds K to e4m3.
Page ids are read into an SGPR so the K buffer descriptors need no waterfall
loop; the causal mask is skipped on tiles whose tokens are all visible; the f16
scale pair is built with one u24 multiply; and a depth table for S8H4 batches
3..16 is selected through geometry(..., nvfp4=True) so MXFP4 work maps do not
move. The e4m3 table and widen helpers are module-level so the prefill scorer
can reuse them.
Co-Authored-By: Claude <noreply@anthropic.com>
Extends the existing decode index-score op test with the NVFP4 scorer. No pytest
in the added code: `check_nvfp4_*` functions run through the file's own runner,
like the MXFP4 checks next to them.
- A torch oracle (e2m1 x e4m3 scale, unrounded and rounded to e4m3) and a torch
NVFP4 page writer, since this change has no NVFP4 write kernel.
- A minimal matrix, both k_round values and both Q dtypes: oracle match on page
and tile boundaries, ragged and empty requests; bit-exact on exact-sum data;
tail pages; scale range and saturation; the NVFP4 depth table; e4m3 Q equals
bf16 Q on the exact arm; split vs single-region views; repeat launches
bitwise identical; the default k_round is "e4m3"; modes do not share a
compiled kernel; rejections.
- An @benchmark table (us, TFLOPS, TB/s, err per k_round) in the decode sweep.
The wider regression matrices stay out of the test file.
Co-Authored-By: Claude <noreply@anthropic.com>
ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm
vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all
All standard extended tests (excludes ci:atom_full)
Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7813 --add-label <label>
One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.
PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.
Reviews:
A reviewer is usually assigned within one working day after the PR is ready for review (see Code Review Process). If nobody has responded for about a week, mention the area contact listed in CONTACTS.md. Want to help review? See REVIEWERS.md.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The NVFP4 predicate accepts inputs with k_round=None that its scorer rejects, risking invalid dispatch, and the work-map rejection test never reaches the intended validation.
Adds NVFP4 decode scoring for the MiniMax-M3 lightning indexer on gfx950, alongside the existing MXFP4 scorer, reusing its work-map machinery and score layout.
Changes:
Implements default e4m3-rounded and exact NVFP4 scoring modes with format validation and tuned launch geometry.
Adds lazy exports for the scorer and support predicate.
Extends correctness checks and performance sweeps, and switches test architecture checks to get_gfx_runtime().
File
Description
op_tests/test_flydsl_minimax_m3_index_score.py
Adds NVFP4 reference generation, correctness checks, and benchmarks.
…ork-map test non-vacuous
index_score_supported_nvfp4 passed k_round=None straight to the shared validator,
which reads None as an MXFP4 cache, so the predicate said True for arguments that
score_flydsl_nvfp4 rejects. Return False unless k_round is one of NVFP4_ROUNDINGS.
The tuned-work-map refusal check fed NVFP4 scale views to the MXFP4 scorer, which
refused them on shape before looking at the work map. Use real MXFP4 views, check the
shared map is accepted and the tuned one is refused with a message naming work_map.
Host-side and test-only; kernel outputs are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The benchmark’s shared/MXFP4 work map causes both candidates to bypass the NVFP4 depth-table optimization, so their timings do not represent the default scorer’s launch.
The benchmark builds a shared/MXFP4 work map, so _validate_metadata retains the shared geometry when it differs from the NVFP4-tuned default, such as S=8, H=4, B=8 at 64K context. Both benchmark candidates therefore bypass the new depth-table optimization and report timings for a different launch than the default scorer. Build the map with nvfp4=True; if shared-map timings are also needed, report them as separate candidates.
bench_score_nvfp4 passed a shared (MXFP4) work map, which score_flydsl_nvfp4
keeps as-is, so the benchmark never ran the NVFP4 depth-table launch that the
default scorer picks (S8H4, batch 3..16). Build the map with nvfp4=True.
Co-Authored-By: Claude <noreply@anthropic.com>
Re the "previously missed issue" in Copilot's review of cf80410 (bench_score_nvfp4 builds a shared work map; no inline thread, so answering here).
Valid, fixed in 80ff1e7: the benchmark now calls build_work_map(seq, mb, S, H, nvfp4=True), so both candidates launch the NVFP4 depth-table geometry that the default scorer picks. _validate_metadata keeps a shared map's geometry when one is passed (kernel.py:2072-2082), so the old benchmark timed a different grid for S8H4.
Measured on one MI355X (median of 5 x 10 iters, B8 S8 H4, uniform ctx, shared map vs nvfp4=True map):
ctx 64K: e4m3 11.10 us (344 rows) vs 9.15 us (512 rows); exact 11.79 vs 9.63 us
ctx 128K: e4m3 15.30 us (688 rows) vs 14.44 us (1024 rows); exact 16.57 vs 15.41 us
I did not add the shared-map timing as a separate candidate (the depth table only changes S8H4 batches 3..16; MXFP4 and other shapes are identical either way). Say if you want it.
python3 op_tests/test_flydsl_minimax_m3_index_score.py --ctx-len (decode checks and benchmarks only) exits 0 with the change; black and ruff 0.16.0 clean. Not run here: the prefill perf sweep (needs the CK submodule to JIT-build module_msa_sparse_attention, absent in my tree; CI has it).
The two earlier threads (k_round predicate, work-map rejection test) were already fixed in cf80410 and are unchanged by this commit.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The new gfx950 MFMA paths and LDS lookup decoding require human review of hardware correctness and performance evidence, as no head-matched runtime report is supplied and GPU validation is unavailable in this environment.
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (5)
State in the description why NVFP4 is needed despite the numbers. The only consumer is draft ATOM#3310; its GSM8K shows nvfp4 at or below MXFP4 (TP2 strict-match 0.8916 vs 0.9030), and this PR's own table has the NVFP4 arms 12-15% slower than score_flydsl_fp4 with 72 vs 68 B/token. The reason is that NVFP4 is MiniMax-M3.1's native fp4 KV/index format and the benchmark rules do not accept a non-native fp4 cache; say that, so the five-PR series reads as required rather than as a slower duplicate of the MXFP4 path.
Fix the NVFP4 module docstring, it describes only one of the three arms.minimax_m3_index_score.py:50-53 says K is widened at cvt(scale=1.0) and Q "is widened to bf16"; the e4m3-Q arms run the fp8 MFMA (k_round="e4m3") or the f16 MFMA ("exact") with 2^120 / 2^8 undo scales. Same in the build_index_score docstring ("q_fp8 (MXFP4 only)"). Also :56 "scale bytes must be positive": zero is accepted and tested, say "non-negative, not 0x7F".
Correct the depth-table comment.minimax_m3_index_score.py:448 says the shared table gives "depth 3 for batches 3..16"; _SERVED_DECODE_DEPTH[(8, 4)] (:441) gives depth 2 for batches 3-4 and depth 3 only for 5..16. In the test, exact_sum_case says scales are in [0.5, 2) for bytes 0x30..0x47 (that range is [0.5, 4)), and check_nvfp4_bit_exact says it "records how often they differ" (it only asserts).
Fold fp8_dwords_to_f16 into fp8_dwords_to_bf16.minimax_m3_index_score.py:155-187 are the same loop with a different intrinsic and element type; the bf16 helper's own docstring says two copies are a silent-wrong-numbers risk. Take the output dtype as a parameter.
Drop references aiter readers cannot follow, and refresh the self-review SHA. "like the vLLM nvfp4 indexer", "vLLM's _quant4", "the P0 microtest", "regression matrix ... lives outside this file" (kernel docstrings and test :2449-2533, :2860): keep the numeric definition (RNE e4m3, saturate at 448, scale rule) and name the test that pins it. aiter-bot's PR Self-Review still fails because "Reviewed revision" is cf80410, not 80ff1e7.
Copilot's ask (gfx950 MFMA hardware correctness, perf numbers): MI35X CI shard 6 ran the NVFP4 matrix (oracle, bit-exact, tail, saturation, depth table, rejections: all PASS), and the description has a 32-shape before/after table vs vLLM and MXFP4, so that ask is answered. Copilot's earlier k_round=None predicate and vacuous work-map test findings are fixed in cf80410; the NVFP4 work map benchmark finding is fixed in 80ff1e7.
Reviewed at 80ff1e7. Stacked follow-up: #7814 (prefill).
Module and build_index_score docstrings described one arm; spell out the
bf16-Q, e4m3-Q/"e4m3" (fp8 MFMA) and e4m3-Q/"exact" (f16 MFMA) routes, say
scale bytes are non-negative and not 0x7F, fix the depth-table comment
(depth 2 for batches 3-4, depth 3 for 5..16) and drop references to vLLM
internals and an unnamed microtest. Comments only.
Co-Authored-By: Claude <noreply@anthropic.com>
…rences
exact_sum_case scale range is [0.5, 3.75] (bytes 0x30..0x47) and its sum
bound 2^14; check_nvfp4_bit_exact only asserts; replace references to vLLM
internals and an out-of-tree regression matrix with the rule itself and the
tests that pin it. Comments and docstrings only.
Co-Authored-By: Claude <noreply@anthropic.com>
fp8_dwords_to_bf16 and fp8_dwords_to_f16 were the same loop with a
different intrinsic and element type. One helper, fp8_dwords_widen, takes
the output dtype; the e4m3-Q NVFP4 loader drops its duplicated branch.
Generated ISA of all 65 kernels the op test builds is byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Thanks @valarLip. All five items are addressed at head 6e6f677 (3 commits on top of 80ff1e7); the description is updated too.
Why NVFP4: added to the description (Summary): NVFP4 is MiniMax-M3.1's native fp4 KV/index format and the benchmark rules do not accept a non-native fp4 cache; the 12-15% / 72 vs 68 B per token cost is stated next to it.
Docstrings:6e46918. The module docstring now lists the three arms (bf16 Q with the cvt/narrow path and the bf16 MFMA; e4m3 Q + "e4m3" with the LDS table, v_perm_b32 and one fp8 scaled MFMA; e4m3 Q + "exact" with the 2^8 undo scale and the f16 MFMA). build_index_score no longer says "(MXFP4 only)". Scale bytes are described as non-negative (zero allowed), not 0x7F.
Depth-table comment and test comments:6e46918 (kernel) and aece88e (test). The comment now says the shared S8H4 table gives depth 2 for batches 3-4 and 3 for 5..16. exact_sum_case says scales are in [0.5, 3.75] (bytes 0x30..0x47) and I corrected the sum bound next to it (2^12 -> 2^14: 128 x 6 x 3.75 x 3.75 = 10800). check_nvfp4_bit_exact now says it asserts that the data separates the two references.
Fold the f16 helper:6e6f677. One fp8_dwords_widen(raw, lo, dwords, dtype) for bf16 and f16; the e4m3-Q loader's duplicated branch is gone. I renamed it because a dtype parameter on a name ending in _bf16 reads wrong; say if you want the old name back. The final ISA of all 65 kernels the op test builds is byte-identical before and after (diff -rq), and the op test is 153 PASS / 0 FAIL both times on one MI355X.
References readers cannot follow:6e46918 and aece88e. The vLLM / _quant4 / "P0 microtest" / "regression matrix lives outside this file" references are replaced by the rule (RNE e4m3, saturate at 448, clamp to +-448 before the convert because it does not saturate; writer rule scale = RNE e4m3 of min(amax x RN(1/6), 448) per 16 elements, then RNE e2m1) and the tests that pin it (check_nvfp4_matches_oracle, check_nvfp4_scale_range_and_saturation). The self-review "Reviewed revision" is now 6e6f677.
Not re-run: the 183-check wide matrix and the vLLM bit comparison (the fixes touch no kernel numerics, ISA identical); CI's pinned ruff 0.16.0 (I used 0.17.3 locally).
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The gfx950 MFMA and NVFP4 changes require independent GPU validation of correctness, rounding, and performance that this source-only review environment cannot provide.
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (3)
Run the NVFP4 scorer through the reachable launch geometries, as the MXFP4 suite does.build_nvfp4_checks (op_tests/test_flydsl_minimax_m3_index_score.py:2862) only uses the F*/L*/causal_straddle/ragged_zero cases (B <= 4) plus one S8H4 B8 tuned-map case. So for NVFP4 the token split at tw2, ppw4 and the non-served ppw2/ppw4 nt2 arms never run. Those are the arms that page through the new SGPR page id, the causal-skip and the e4m3 LDS table. Add check_nvfp4_every_geometry_matches_oracle over the non-CP GEOM_POINTS x QDTS x K_ROUNDS, reusing check_mxfp4_every_geometry_matches_oracle (:2165) with nvfp4_cache_of. While there, also call expect_value_error(... run_nvfp4 ...) on the int8 and wide-stride scale cases (:2818-2825), which only check the predicate today.
Fix the comments the 11-15 rewrite got wrong.
minimax_m3_index_score.py:311-313: check_nvfp4_matches_oracle is an ATOL/RTOL checkAllclose (check_scores, test :1248). The bit-equality check is check_nvfp4_bit_exact.
:353-354: e2m1 x e4m3 needs six significant bits, not five (1.5 x 1.875 = 101101b).
:2445-2447: re-wrap the broken "and not / 0x7F." docstring line.
:579-581: the geometry docstring says an nvfp4=True map is "only valid for score_flydsl_nvfp4". That is true only for S8H4 batches 5..16; everywhere else the two maps are identical.
build_work_map / work_map_size (:1765, :1818) and score_flydsl_nvfp4 do not say that nvfp4=True (passed through **cp) is how a caller gets the tuned depth table.
Small cleanups.
:954: elif const_expr(nvfp4 and q_fp8) is exactly f16_arm there; use the named flag like every other branch.
nvfp4_single_region_views (test :2534) is single_region_views (:2108) with 4 -> 8; give it a scale-width parameter instead of a second copy.
CI: all 30 checks pass (27 skipped). The premise is accepted: NVFP4 is M3.1's native fp4 index format and is now stated in the description.
Re-reviewed at 6e6f677. Already fixed: the "why NVFP4" paragraph in the description; the three-arm module and build_index_score docstrings and the "non-negative, not 0x7F" wording (6e46918); the S8H4 depth-table comment, exact_sum_case range [0.5, 3.75] with the 2^14 bound, and the check_nvfp4_bit_exact docstring; fp8_dwords_to_f16 folded into fp8_dwords_widen (6e6f677); vLLM / _quant4 / P0 / out-of-tree references removed; self-review SHA refreshed. Copilot's k_round=None predicate, vacuous work-map test and NVFP4 benchmark-map findings stay fixed.
…vfp4=True on the map builders
Comments only plus one flag rename: f16_arm is exactly `nvfp4 and q_fp8` at that branch.
Co-Authored-By: Claude <noreply@anthropic.com>
…ch geometry
Reuses the MXFP4 every-geometry check (token split, ppw 2/4, non-served nt2 arms), checks
ValueError on the int8 and wide-stride scale views, and folds nvfp4_single_region_views
into single_region_views (scale width from the view).
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The new gfx950 numerical paths require domain-reviewer scrutiny because author-reported results lack independent, exact-head GPU validation of correctness and base/head latency.
@valarLip, re-review of 11-15 17:47, point by point. Pushed head is now 9a5d21b (fast-forward of 6e6f677; fixes in 6a9e5a8 and 9a5d21b). The description is updated to match.
NVFP4 scorer only ran B <= 4 cases: fixed in 9a5d21b. check_nvfp4_every_geometry_matches_oracle runs the 10 non-CP GEOM_POINTS x bf16/e4m3 Q x e4m3/exact K (40 checks) through check_mxfp4_every_geometry_matches_oracle, which now takes an optional k_round and then uses nvfp4_cache_of + run_nvfp4. The scorer has no CP shard, so the CP points are left out. The geometries reached for NVFP4 include tw2, tw4, ppw4 and the non-served ppw2/ppw4 nt2 arms. One note: the S8H4 b16 point resolves to ppw 2 for NVFP4 (the tuned table); ppw 3 is only reached with an explicit shared work map, which the existing S8H4 B8 tuned-map check runs. The int8 and wide-stride scale cases in check_nvfp4_rejections now also check that run_nvfp4 raises ValueError. On one MI355X all 40 pass (21 s with first compile) and the full default run passes 193 checks, 0 FAIL. I did not mutation-test the sweep: no kernel code changed in this round.
Comments: all five were right; fixed in 6a9e5a8. check_nvfp4_matches_oracle -> check_nvfp4_bit_exact. "five significant bits" -> six (1.5 x 1.875 = 10.1101b; I also checked every e2m1 x e4m3 product, the maximum is six). The "and not / 0x7F" line is re-wrapped. The geometry docstring now says the two maps differ only at S8H4 batches 5..16 and are identical elsewhere. build_work_map, work_map_size and score_flydsl_nvfp4 now say that nvfp4=True (through **cp) gets the tuned depth table, and that work_map=None resolves it too.
Cleanups: elif const_expr(nvfp4 and q_fp8) is f16_arm at that point, so it now uses the flag (6a9e5a8). nvfp4_single_region_views is gone; single_region_views reads the scale width from the scale view, so no extra parameter is needed (9a5d21b). That differs slightly from the parameter you suggested; say so if you prefer the explicit one.
Not re-measured after this round: ISA and the performance tables (the only kernel change is the f16_arm flag, equal to the expression it replaces, plus comments).
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (3)
Drop the dead bf16 branches left in the f16 arm. Naming f16_arm at minimax_m3_index_score.py:956 left the same redundancy one level down. load_q8_lifted is only called under elif const_expr(f16_arm) (:957), but :932 still says wdt = fx.Float16 if f16_arm else fx.BFloat16, and :960, inside that same branch, builds the MFMA with fx.Float16 if f16_arm else fx.BFloat16. Use fx.Float16 in both places. Also fix the load_q8_lifted docstring (:921): "the four bf16 B-fragments" should say f16.
Give the shared every-geometry check a format-neutral name, not a wrapper.check_nvfp4_every_geometry_matches_oracle (test :2569) only forwards to check_mxfp4_every_geometry_matches_oracle (:2166), and that MXFP4-named function now holds the NVFP4 branch too. Rename it (for example check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None)), call it from both build_mxfp4_checks and build_nvfp4_checks, and delete the one-line wrapper.
Update the PR description to match head. "Changed op tests" says "the MXFP4 checks are unchanged apart from the arch gate". That is no longer true: single_region_views and the every-geometry check are now shared with NVFP4. The Tests paragraph still says "about 20 s" and its matrix list leaves out the 40-check NVFP4 geometry sweep (you measured about 21 s for the sweep alone).
CI: the decode op test passes at head (shard 6, including all 40 new every_geometry_* NVFP4 checks). The only failure is test_flydsl_hgemm.py ("graph replay mismatch"), which does not import this module, so it is unrelated to this PR.
Re-reviewed at 9a5d21b. Already fixed: the NVFP4 sweep over the 10 non-CP GEOM_POINTS x Q dtype x k_round (S8H4 b16 now runs the tuned ppw2 geometry); expect_value_error on run_nvfp4 for the int8 and wide-stride scales (the "k_scale" and "packed" substrings match the real messages); the check_nvfp4_bit_exact, "six significant bits", 0x7F re-wrap, geometry nvfp4 and nvfp4=True-through-**cp docstrings; f16_arm at :956; nvfp4_single_region_views folded into single_region_views. Earlier rounds' items and Copilot's findings stay fixed.
…LDS/barrier comments, restore score_flydsl docstring
load_q8_lifted is only called under f16_arm, so its dtype and the MFMA dtype are plain
Float16. The "no LDS, no barrier" comments now name the fp8 arm's scale table and its
barrier. Final ISA of 56 NVFP4 e4m3-Q kernels is identical before and after.
Co-Authored-By: Claude <noreply@anthropic.com>
@valarLip, re-review of 11-15 21:16, point by point. Pushed head is now 499b0e0 (fast-forward of 9a5d21b): c4c66f3 (kernel and comments) and 499b0e0 (test). The description is updated to match.
Dead bf16 branches in the f16 arm: fixed in c4c66f3. load_q8_lifted is only called under elif const_expr(f16_arm), so the fragment dtype and the MFMA dtype are now plain fx.Float16, and its docstring says "four f16 B-fragments". The generated code is unchanged: I compiled 56 NVFP4 e4m3-Q kernels (both k_round, S1H1/S4H4/S8H4/S16H4, token split, ppw 1..4, spread; 28 of them the f16 arm) with no GPU before and after, and the final ISA is byte-identical for all 56 by sha256.
Format-neutral name for the every-geometry check: fixed in 499b0e0. The shared function is check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None), called from both build_mxfp4_checks and build_nvfp4_checks; the one-line wrapper is deleted. The check list is unchanged: 100 MXFP4 and 93 NVFP4 checks (40 of them the NVFP4 sweep), same names before and after.
Description: updated. "Changed op tests" now says the MXFP4 checks check what they did before and that single_region_views and the every-geometry check are shared; the Tests paragraph says about 20 to 40 s and lists the 40-check NVFP4 geometry sweep (about 21 s on its own); the per-file line counts are recomputed (+614/-65 for the kernel file, +528/-29 for the test).
Two more things I fixed in c4c66f3, from our own review of this head: the comments that said "no LDS, no barrier" now name the fp8 arm's 1 KiB scale table and its barrier, and score_flydsl's public docstring is back to describing its own arguments instead of pointing at the private _score_impl.
On one MI355X the default run of the op test passes 193 checks, 0 FAIL, before and after this round. Not re-measured: the performance tables (the kernel change is the fx.Float16 constants, which do not change the ISA, and comments), and CI's ruff 0.16.0.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The new gfx950 NVFP4 and MFMA paths require native correctness and performance validation, but no head-matched report was supplied and this environment cannot run the GPU tests.
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (3)
Compute the f16-arm scale pair once per byte, as the bf16 arm does.minimax_m3_index_score.py:1180-1184 calls e4m3_scale_raw_f16x2(w16) twice and e4m3_scale_raw_f16x2(w16 >> 8) twice; the else branch right below (:1187-1189) already does s_lo/s_hi then [s_lo, s_lo, s_hi, s_hi]. Use the same shape in both arms.
Split the _validate_metadata NVFP4 guard so its message names the real problem.:2043 raises "nvfp4: k_round must be one of ..." also when k_round is valid and k_scale is None. Raise a separate "nvfp4 needs k_scale" error for that case.
Say "non-negative" everywhere the scale contract is stated. The module docstring (:54) and score_flydsl_nvfp4 say non-negative (zero allowed, and zero is tested), but the PR description ("Scale bytes must be positive e4m3"), e4m3_to_f32's docstring (:301, "scales are positive") and the test comment at :2688 ("0x00..0x7E: scales are positive") still say positive.
Optional, not blocking: nvfp4 is a bool in geometry/_decode_depth_table and the K-rounding string in build_index_score/_get/_validate_metadata; a distinct name for the geometry flag would read better. The test's K_ROUNDS (:2474) duplicates kernel.NVFP4_ROUNDINGS.
CI: 31 pass, 27 skipped, no failures.
Re-reviewed at 499b0e0. Already fixed: the dead fx.BFloat16 branches in the f16 arm (load_q8_lifted and its MFMA are now plain fx.Float16, docstring says f16); one format-neutral check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None) called from both builders, wrapper deleted; the description's "Changed op tests" and Tests paragraphs (shared helpers, 20-40 s, the 40-check sweep, line counts +614/-65 and +528/-29 match the diff). Earlier rounds' items and Copilot's findings stay fixed.
Closing as landed: this PR's code is already on main. #7814 was stacked on this branch (its head contains 499b0e0f45), so its squash merge dad1b66282 brought in minimax_m3_index_score.py and test_flydsl_minimax_m3_index_score.py exactly as they are at this head; they are byte-identical to main now.
@valarLip, thanks for the reviews. Three items from your last round (06:20, at 499b0e0f45) did not make it into main, and I am not following them up in this PR: (1) hoisting the f16-arm scale pair to once per byte, (2) a separate "nvfp4 needs k_scale" error in _validate_metadata, (3) "non-negative" wording in the e4m3_to_f32 docstring, the LUT_ENTRIES comment and the test comment. I checked (1) on CPU: the generated ISA is identical before and after (112 kernels, same hashes), so it is a readability change only. I have a tested patch for all three (2 files, +21/-16, cherry-picks cleanly onto current main); if you want it, I will send it as a small follow-up PR against main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the decode block scorer for the MiniMax-M3 lightning indexer over an NVFP4 index cache, next to the existing MXFP4 one (
score_flydsl_fp4): same work-map machinery, same score layout, new page format. Exportsminimax_m3_index_score_nvfp4_flydslandminimax_m3_index_score_nvfp4_flydsl_supported. Only gfx950 (MI355X). A follow-up PR adds the prefill scorer and builds on this one.Why NVFP4 at all, when it is slower than MXFP4. NVFP4 is MiniMax-M3.1's native fp4 KV/index format, and the benchmark rules do not accept a non-native fp4 cache, so the MXFP4 scorer cannot serve the model as it is. The cost is measured in the Performance section below: the NVFP4 arms take 12-15% more time than
score_flydsl_fp4(geomean e4m3/mx 1.125, exact/mx 1.150) and 72 vs 68 B per token.Page format
One NVFP4 index page is 9216 B: 128 tokens x 64 B of e2m1 codes (two 4-bit codes per byte, low nibble = even element) = 8192 B, followed by 128 tokens x 8 B of e4m3 scales = 1024 B. One scale per 16 elements, no per-tensor scale. Q is e4m3 (bf16 is accepted). The scorer reads the two regions as separate views (
k_data [pages, 128, 64]uint8 andk_scale [pages, 128, 8]uint8, each with its own page stride), so a single[pages, 9216]allocation works. Scale bytes must be positive e4m3 and not 0x7F; contents are a caller contract, shapes, dtypes and strides are validated.Numerics:
k_roundk_round="e4m3"(default): K = e4m3_rn_satfinite(e2m1 x scale), the rounding vLLM's NVFP4 indexer applies. With an e4m3 Q it runs on the fp8 MFMA: each CTA builds a table of the 8 e4m3 values of every scale byte in LDS and widens a K dword withv_perm_b32. Scores are bit-identical to vLLM's NVFP4 decode indexer (60 cases, 208,999 live slots, 100% bit-equal on one shared page buffer, live-slot sets identical; table below).k_round="exact": K = e2m1 x scale, not rounded. The product is exact in f16 and so is an e4m3 Q, so with an e4m3 Q the K dwords are converted straight to f16 pairs and scored on the f16 MFMA; a bf16 Q uses the bf16 MFMA. This is not bit-identical to vLLM. Measured top-k block overlap with vLLM's scores (random-normal K and Q quantized to NVFP4, so a synthetic distribution; rows with at least k live blocks; the e4m3 arm is 1.0 everywhere):What changes
aiter/ops/flydsl/kernels/minimax_m3_index_score.py(+614/-65): the NVFP4 arms, the e4m3 table and widen helpers (module level, so the prefill scorer reuses them), validation,score_flydsl_nvfp4,index_score_supported_nvfp4. Page id in an SGPR (no waterfall loop in the K loads), causal mask skipped on tiles whose tokens are all visible, f16 scale pair from one u24 multiply, and an NVFP4 depth table for S8H4 batches 3..16 selected withgeometry(..., nvfp4=True)/build_work_map(..., nvfp4=True)so MXFP4 work maps do not move. The table gives, on S8H4 (same process, e4m3 arm / exact arm, depth table vs shared map): batch 8 ctx 64K 0.83 / 0.83, batch 16 ctx 32K 0.79 / 0.79, batch 8 ctx 128K 0.95 / 0.92, batch 2..4 unchanged (1.00 +- 0.01).aiter/ops/flydsl/__init__.py(+10): the two lazy exports.op_tests/test_flydsl_minimax_m3_index_score.py(+528/-29): see Tests.No csrc, no tuned CSV: the
aiter-config-shaperules do not apply (the diff has no files underaiter/configs).Performance (decode, vs vLLM)
One MI355X, same process, interleaved, 7 repetitions, two independent runs (r1/r2 columns), 32 shapes: S = query tokens per request, H = index heads, B = batch, ctx = context.
mxis aiter's MXFP4 scorer,vllmis the MiniMax-M3 NVFP4 decode indexer of our vLLM tree (models/minimax_m3/amd/ops/nvfp4_index_decode, unmodified). TB/s is effective K-page bandwidth (live pages x 128 x 72 B per token for NVFP4, x 68 B for MXFP4, over the time); K is re-read from the same buffer every iteration, so the small shapes are warm-cache numbers. Ratios are by mean time (below 1 = faster than vLLM). The nvfp4 columns do not include the work-map build, as for MXFP4.exact/vLLM time ratio: geomean 0.823, worst 0.940, best 0.648, slower than vLLM in 0 of 32 shapes
e4m3/vLLM time ratio: geomean 0.805, worst 0.905, best 0.619, slower than vLLM in 0 of 32 shapes
shapes where exact is slower than vLLM (mean of the two runs):
none
shapes where e4m3 is slower than vLLM (mean of the two runs):
none
Shapes where an NVFP4 arm is slower than vLLM: none in this table. Against aiter's own MXFP4 scorer the NVFP4 arms are slower, as expected for 72 B vs 68 B per token and the extra decode work: geomean e4m3/mx 1.125, exact/mx 1.150 by time.
Raw table: 32 shapes, two runs
Tests
python3 op_tests/test_flydsl_minimax_m3_index_score.py(about 20 to 40 s on one MI355X; exits non-zero on any failed check): the existing MXFP4 checks, then the NVFP4 ones, then the sweep tables. The NVFP4 part follows the aiter op-test shape: no pytest in the added code (check_nvfp4_*functions run by the file's own runner, as for MXFP4),@benchmarkwithrun_perftest,checkAllclose, TFLOPS and TB/s per candidate, a markdown table, andget_gfx_runtime()gating inmain(). The torch oracle dequantizes the cache itself and a torch writer produces the pages, because this change has no NVFP4 write kernel. Matrix: oracle match on page and tile boundaries, ragged and empty requests; bit-exact on exact-sum data; tail pages; scale range and saturation; the depth table; e4m3 Q equals bf16 Q on the exact arm; split vs single-region views; repeat launches bitwise identical; the defaultk_roundis "e4m3"; compiled kernels are not shared between modes; rejections; bothk_roundvalues and both Q dtypes; and a sweep through every non-CP launch geometry the selector can reach (10 geometry points x bf16/e4m3 Q x e4m3/exact K, 40 checks, about 21 s on its own, run bycheck_fp4_every_geometry_matches_oracle, the same function the MXFP4 builder calls). A wider regression matrix (183 NVFP4 decode checks, hundreds of shapes, bit comparison against vLLM) was run outside the PR on this tree; it is not part of the diff.Checked on this tree with flydsl 0.5.0 (the version
requirements.txtpins), gfx950:origin/main.Dependencies and scope
flydsl==0.5.0 as pinned, gfx950 only. Not in this PR: the NVFP4 index write kernel (the tests write pages with torch), the prefill scorer (next PR), ATOM integration.
PR Self-Review
Reviewed revision
499b0e0 (head after the second 11-15 review round; the full sha is the PR head)
Behavior and intent
Adds
score_flydsl_nvfp4andindex_score_supported_nvfp4: decode block scoring over an NVFP4 index cache (9216 B pages: e2m1 codes plus one e4m3 scale per 16 elements). Before this PR aiter could only score the MXFP4 cache, so the MiniMax-M3 NVFP4 index cache had no aiter scorer. The defaultk_round="e4m3"arm is bit-identical to vLLM's NVFP4 decode indexer;k_round="exact"keeps K unrounded and is not bit-identical. The predicate returns False for anyk_roundthe scorer rejects, includingNone.Validation and unverified scope
Run:
python3 op_tests/test_flydsl_minimax_m3_index_score.pyon one MI355X (gfx950, flydsl 0.5.0): exit 0 on the head above. Re-checked after the review fixes: 76 sha256 hashes of MXFP4 decode and prefill outputs identical toorigin/main, and the NVFP4 decode output hashes (both arms, both Q dtypes) identical to the pre-fix tree. Run before the review fixes, on the same kernel code: 183 NVFP4 decode checks and 60 cases / 208,999 live slots 100% bit-equal to vLLM for the e4m3 arm (not re-run, the fixes touch no kernel code). Unverified: gfx942 and other flydsl versions (gfx950 only by design), real selector/model data (synthetic random-normal K and Q), the "exact" arm's agreement with vLLM beyond the top-k overlap table. The hgemm CI failure (test_flydsl_hgemm.py) is outside this diff.Re-run at the revision above after the review fixes (3 commits on 80ff1e7: docstrings and comments, test comments, and one refactor folding
fp8_dwords_to_f16intofp8_dwords_widen(raw, lo, dwords, dtype)):python3 op_tests/test_flydsl_minimax_m3_index_score.py --ctx-len 8192on one MI355X (gfx950, flydsl 0.5.0): exit 0, 153 checks PASS, 0 FAIL, before and after the refactor (at 6e6f677).f16_armnamed flag, docs ofnvfp4=Trueon the map builders; 9a5d21b: the NVFP4 every-geometry sweep,expect_value_erroron the int8 and wide-stride scale cases,single_region_viewsshared): the default run on one MI355X passes 193 checks, 0 FAIL, including the new sweep of 40 checks (10 non-CP geometry points x bf16/e4m3 Q x e4m3/exact K); the sweep alone takes about 21 s. The kernel change in that round is only thef16_armflag (equal to the expression it replaces at that branch) and comments; the ISA and the performance tables were not re-measured after it.fx.Float16, the LDS/barrier comments,score_flydsl's public docstring; 499b0e0: one format-neutralcheck_fp4_every_geometry_matches_oraclefor both builders instead of an NVFP4 wrapper): the final ISA of 56 NVFP4 e4m3-Q kernels (bothk_round, 28 of them the f16 arm), compiled without a GPU before and after, is byte-identical by sha256; the default run of the op test on one MI355X passes 193 checks, 0 FAIL, before and after (100 MXFP4 and 93 NVFP4 checks, same names). The performance tables were not re-measured.FLYDSL_DEBUG_DUMP_ASM) of all 65 kernels the test builds, including the NVFP4 e4m3-Q arms for bothk_roundvalues, is byte-identical before and after (diff -rq).PR-specific question
Before this PR every NVFP4 check in the changed test fails at the first call (
score_flydsl_nvfp4does not exist). Beyond that, the checks pin things a plain oracle match would not: bit-exact scores on exact-sum data, tail pages, scale range and saturation, empty and ragged requests, e4m3 Q equals bf16 Q on the exact arm, repeat launches bitwise identical, thek_rounddefault and the predicate fork_round=None, and rejection of bad shapes/strides/dtypes. The tuned work-map check fails if the MXFP4 exact-grid check in_validate_mapis removed. No canonical test covers the NVFP4 page format; the existing MXFP4 test only rejects such a cache.Changed op tests
op_tests/test_flydsl_minimax_m3_index_score.py(+528/-29): the canonical index-score test, extended in place (no new file) with the NVFP4 oracle, correctness matrix and benchmark; the MXFP4 checks check what they checked before, apart from the arch gate (nowget_gfx_runtime());single_region_viewsand the every-geometry check (check_fp4_every_geometry_matches_oracle, called by both the MXFP4 and the NVFP4 builder) are now shared with the NVFP4 side. It is not redundant with any other test: no other test builds NVFP4 pages or calls the NVFP4 scorer.🤖 Generated with Claude Code